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ABSTRACT 


This  paper  describes  insights,  procedures,  and  limitations 
involved  in  the  semi-automatic  solution  of  large-scale  computer 
simulation  problems.  It  utilizes  the  experience  gained  in 
solving  test  problems  by  both  a  human  analyst  and  an  automatic 
OPTIMIZER  program.  Particular  attention  is  paid  to  those  tasks 
performed  by  the  man  but  not  the  machine  and  those  tasks  best 
done  by  the  machine.  Guidelines  are  suggested  for  incorporating 
some  of  the  human  problem-solving  processes  as  a  first  step  toward 
interactive  semi-automatic  procedures. 


ADMINISTRATIVE  INFORMATION 

This  effort  was  carried  out  by  the  Technology  Development 
Group  of  the  Operations  Research  Division.  It  was  sponsored 
by  the  Naval  Analysis  Programs  of  the  Office  of  Naval  Research, 
Code  462,  under  program  elements  65104N,  61102N,  and  65152N, 
and  performed  under  work  unit  number  1-1863-014. 
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1.  Background 


Many  of  the  military  operations  research  problems  confronting 
decision  makers  today  are  either  too  expensive  for  experimental 
solution  or  too  complicated  for  analytical  treatment.  A  solution  to 
these  complex  problems  is  frequently  sought  through  computer  simu¬ 
lation.  However,  computer  simulation  is  subject  to  such  practical 
constraints  as  reporting  deadlines  and  approved  budget  expenditures 
(investment  in  computer  processing  and  analyst  time).  These  general 
resource  guidelines  establish  a  basic  framework  which  limits  the  number 
of  alternative  problem-solving  approaches  that  may  be  pursued  by  the 
analyst. 

The  quest  for  a  solution  to  a  complex  problem, such  as  determining 
the  optimal  tactics  to  be  exercised  in  a  naval  engagement,  begins 
with  an  analyst  making  judgments  as  to  possible  "good"  tactics.  Then, 
using  computer  simulation,  he  examines  the  results  of  these  tactics. 

On  the  basis  of  information  gained  from  the  simulation,  he  may  determine 
other  candidate  tactics.  This  process  may  be  repeated  a  number  of  times 
When  the  analyst  considers  that  no  better  results  can  be  obtained  or,  as 
is  more  likely,  runs  out  of  time,  he  selects  the  "best"  tactics  in  terms 
of  some  measure  of  effectiveness. 

In  far  too  many  instances,  this  search  for  an  improved  solution 
degenerates  into  a  trial-and-error  process.  Recognition  of  this 
situation  gave  rise  to  the  concept  of  a  computer  simulation  OPTIMIZER.'*' 
The  OPTIMIZER  is  an  executive  FORTRAN , computer  program  which  can  easily 
be  interfaced  with  a  simulation  to  conduct  an  adaptive,  mathematically- 
disciplined  search  for  an  optimal  solution. 

The  development  of  the  OPTIMIZER  was  based  on  a  "black  box"  view 
of  computer  simulation.  That  is,  the  simulation  was  regarded  as  a 
"black  box"  in  which  the  values  of  input  parameters  (also  called  input 
variables)  are  combined  in  some  manner  to  produce  output  parameters. 


1.  Smith,  Dennis  E. ;  "An  'OPTIMIZER'  for  Use  in  Computer  Simulation: 
Background  and  Design  Concepts,"  HRB-Singer,  Inc.,  State  College,  Pa. 
16801,  July,  1970,  Report  No.  4352.11-R-l 
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The  input  parameters  are  of  two  types :  (1)  controllable  factors ,  and 

(2)  uncontrollable  factors.  Controllable  factors  are  those  Input 
parameters  which  may  be  directly  controlled  by  the  decision  maker  in 
the  "real  world";  uncontrollable  factors  are  those  input  parameters 
over  which  the  decision  maker  has  no  direct  control.  For  example,  in 
a  destroyer  screening  problem  the  controllable  factors  are  those 
describing  escort  tactics  (such  as  positions,  bearings,  speeds,  etc.). 
Uncontrollable  factors  are  those  describing  enemy  tactics  and  sea 
environment. 

2 

Information  gained  from  studies  with  a  prototype  OPTIMIZER 
verified  the  potential  usefulness  of  the  basic  OPTIMIZER  concept 
but  pointed  out  the  need  for  incorporating  several  additional  features 
into  the  existing  computer  program.  In  addition,  a  search  algorithm 

3 

based  on  Response  Surface  Methodology  (RSM) ,  which  is  a  blending  of 
the  statistical  techniques  of  experimental  design  and  regression 
analysis,  was  identified  as  tending  to  offer  the  greatest  payoff  in 
application. 

Because  of  its  promise,  an  RSM  search  algorithm  was  selected  as 
the  basis  of  a  modified  OPTIMIZER  containing  additional  features 
suggested  from  experience  with  the  prototype.  The  OPTIMIZER,  as 
currently  designed,  is  applicable  to  problems  of  determining  those 
values  of  continuous  (or  approximately  continuous)  controllable 
factors  which  produce  an  optimum  value  of  one  chosen  output  parameter 
of  interest.  This  parameter  is  used  as  a  measure  of  effectiveness  (MOE) 
for  the  simulation.  Linear  constraints  on  the  controllable  factors  are 
permitted. 


2.  Smith,  Dennis  E. ;  "An  ’OPTIMIZER'  for  Use  in  Computer  Simulation: 
Studies  with  a  Prototype,"  HRB-Singer,  Inc.,  State  College,  Pa.  16801, 
September,  1971,  Report  No.  4352.11-R-2 

3.  Box,  G.E.P.  and  Wilson,  K.B.;  "On  the  Experimental  Attainment  of 
Optimum  Conditions,"  Journal  of  the  Royal  Statistical  Society  (Series  B) , 
Vol  13,  p.  1,  1951 
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2.  Study  Framework 
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The  ASW  Programs  Surface  Ship  Engagement  Model  (APSURF)  was  chosen 
as  the  simulation  for  test  and  evaluation  of  the  OPTIMIZER.  The  general 
approach  was  first  to  solve  an  anti-submarine-warfare  (ASW)  problem  using 
an  analyst  and  manual  interface  with  the  APSURF  model  in  a  conventional 
manner.  The  same  problem  was  then  solved  independently  by  the  OPTIMIZER 
program.  Comparison  of  the  problem-solving  methodology  and  the  results 
obtained  in  the  two  problem  solutions  was  used  as  the  basis  for  OPTIMIZER 
evaluation. 

After  this  evaluation  had  been  made,  the  scope  of  the  effort  was 
extended  by  using  this  specific  encounter  as  a  basis  for  further 
investigations  into  the  more  general  area  of  automated  problem  solving. 
The  basic  approach  was  to  examine  the  systematic  techniques  employed  by 
the  human  analyst  in  his  solution  of  the  problem,  paying  particular 
attention  to  those  aspects,  categorized  as  "sub-analyses,"  which  the 
analyst  performed  between  certain  key  runs.  Although  performed  in  the 
analyst's  head  or  on  a  piece  of  scratch  paper  (with  possible  assistance 
of  prepared  graphs) ,  these  tasks  proved  to  be  complete  analyses  in 
themselves.  It  was  hoped  that  identifying  these  subtasks  would  provide 
a  foundation  for  incorporating  them  into  the  OPTIMIZER  type  of  automated 
process. 

Possible  ways  of  combining  manual  and  automated  processes  have  also 
been  investigated.  This  aspect  of  the  effort  involves  solving  part  of 
the  problem  manually  and  then  turning  the  partially  solved  problem  over 
to  the  automated  process.  Such  an  effort  encompasses  both  establishing 
the  sequence  of  the  solution  techniques  and  also  developing  methods  of 
using  human-processed  feedback  information  to  modify  the  automated 
process.  The  sequence  of  solution  techniques  was  experimented  with  in 


4.  Flum,  R.S.;  SAO  (Systems  Analysis  Office  of  PM-4)  Report  69-16, 
"APSURF  Mod  1  -  ASW  Programs  Surface  Ship  Engagement  Model,"  January, 
1970  (Unclassified) 

Abstract  (AD  881-384L) 

Vol.  1,  Part  1  (Sec.  1-5)  (AD  881-385L) 

Vol.  1,  Part  1  (Sec.  6)  (AD  881-386L) 

Vol.  2,  Part  1  (Sec.  1-3)  (AD  881-387L) 

Vol.  2,  Part  1  (Sec.  4  &  Appendixes)  (AD  881-388L) 
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the  actual  OPTIMIZER  evaluation.  The  development  of  human-processed 
feedback  information  will  be  a  future  effort  and  should  allow  the  analyst 
to  override  the  automated  pro cess -selection  techniques  and  direct  the 
machine  problem  solver  to  use  a  different  technique  when  necessary. 

This  paper  will  generally  describe  the  evaluation  effort**  already 
undertaken  and  the  analysis  of  specific  situations  which  form  the  basis 
for  recommending  future  efforts  in  the  man-machine  process  of  semi- 
automated  problem  solving  with  large-scale  computer  simulation  models. 

The  subsequent  sections  will  cover,  in  order,  the  following  areas: 

•  Test  Problem  Description 

•  Results  of  Evaluation  Test 

•  Overview  of  Analyst's  Methodology 

•  Implications  to  Semi -Automated  Problem  Solving 


5.  Smith,  Dennis  E.  and  Storck,  C.  Edward;  "Application  of  an  'OPTIMIZER' 
Computer  Program  to  a  Large-Scale  Naval  Simulation  Model,"  HRB-Singer,  Inc., 
State  College,  Pa.  16801,  June  1973,  Report  No.  4352. llR^ 
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3. 


Test  Problem  Description^ 

The  test  problem  was  designed  primarily  for  the  evaluation  and 
hence  was  not  intended  to  represent  an  actual  ASW  study  problem.  The 
scenario  involved  a  task  force  moving  north  at  a  constant  speed, 
protected  by  three  escorts.  The  submarine  threat  always  approached 
from  some  northerly  direction.  The  escorts  were  each  assigned  a 
patrol  area  generally  forward  of  the  task  force.  The  first  patrol 
area  was  northeast  of  the  task  force,  the  second  due  north,  and  the 
third  northwest,  as  illustrated  in  Figure  1. 

The  measure  of  effectiveness  (MOE)  for  the  test  problem  was  the 
product  of  the  probability  of  detection  of  the  submarine  and  the  average 
distance  from  the  task  force  to  the  submarine  when  a  detection  occurred. 

The  evaluation  was  based  on  a  comparison  of  the  OPTIMIZER'S  and  the 
analyst's  maximization  process  and  on  the  value  of  the  MOE. 

The  maximization  was  accomplished  by  manipulating  the  values  of  29 
controllable  variables,  subject  to  constraints.  The  29  variables  defined 
(1)  the  two  ranges  from  the  task  force  to  the  centers  of  the  first  and 
second  patrol  areas;  (2)  the  speeds  of  escorts  #1  and  #2;  (3)  the  bearing 
of  the  center  of  patrol  area  #1  from  the  task  force;  and  (4)  six  ranges  and  six 
bearings  from  the  centers  of  each  of  patrol  areas  #1  and  #2.  These 
ranges  and  bearings  defined  six  points  for  the  respective  escorts  to 
patrol.  These  patrol  points  were  traversed  in  a  specified  order 
defining  a  patrol  pattern  for  the  escort.  Each  complete  cycle  was 
repeated.  The  patrol  pattern  for  escort  #3  was  a  mirror  image  of 
that  for  escort  #1. 


6.  Mandelbaum,  Jay;  NSRDC  CMD  Technical  Note,  CMD-17-72,  "Analyst's 
Test  Problem  Used  to  Evaluate  HRB  Singer  Optimizer  for  Large-Scale 
Computer  Simulation  Models,"  May  1972 
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t  Problem  Scenario 
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Results  of  Evaluation^ 

The  first  objective  of  the  study  described  in  this  paper  was  to 
provide  an  evaluation  of  the  OPTIMIZER,  based  on  direct  comparison  of 
the  solution  obtained  by  the  analyst  and  that  found  by  the  OPTIMIZER- 
controlled  simulation  runs.  Evaluation  of  the  overall  study  involved 
three  primary  OPTIMIZER  search  attempts. 

Examination  of  these  three  search  cases  revealed  that  the 
completely  automated  OPTIMIZER  search  did  not  fare  well  compared 
to  the  analyst's  search.  However,  a  good  solution  involving  only  a 
small  number  of  runs  was  obtained  by  the  OPTIMIZER  when  it  was 
allowed  to  start  its  search  after  a  preliminary  analyst  search.  In 
fact,  the  value  of  the  MOE  produced  by  the  OPTIMIZER  using  this  prelim¬ 
inary  analyst  search  was  the  highest  value  obtained  among  all  the 
simulation  runs  made  by  either  the  analyst  or  the  OPTIMIZER. 

Starting  from  a  "base  case"  or  starting  point  chosen  by  the 
analyst  and  using  the  29  controllable  factors  described  previously, 
the  OPTIMIZER  search  located  an  MOE  value  which  was  decidedly  inferior 
to  the  MOE  values  found  by  the  analyst.  (See  Figure  2.)  Although  the 
OPTIMIZER  used  all  29  factors  in  conducting  its  search,  the  analyst 
essentially  reparameterized  the  problem  so  that  he  had  to  manipulate 
only  a  small  number  of  these  factors  in  his  optimization  attempt.  An 
OPTIMIZER  search  was  then  conducted  using  the  six  reparameterized 
factors.  The  OPTIMIZER'S  performance  using  these  six  factors  was 
significantly  better  than  its  performance  using  the  original  29 
factors.  However,  by  run  No.  30,  it  became  apparent  that  the  analyst 
was  still  outperforming  the  OPTIMIZER.  (See  Figure  3.) 

Because  of  the  OPTIMIZER'S  relative  lack  of  success,  it  was 
hypothesized  that  (1)  the  relationship  of  the  MOE  to  the  controllable 
factors  might  be  quite  complex  in  the  starting  region  for  the  search, 
and  (2)  through  his  intuitive  understanding  of  the  problem  the  analyst 


7.  Mandelbaum,  Jay;  NSRDC  CMD  Technical  Note,  CMD-2-73,  "HRB  Singer 
Optimizer  Performance  on  an  Evaluation  Test  Problem,"  January  1973 
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Figure  2  -  OPTIMIZER  and  Analyst  Run  Histories  (OPTIMIZER  Search  No.  1) 
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Run  Number 

Figure  3  -  OPTIMIZER  and  Analyst  Run  Histories  (OPTIMIZER  Search  No.  2) 


managed  to  guide  his  search  around  these  complexities,  while  the 
OPTIMIZER  could  not.  Therefore,  a  better  procedure  might  be  for  the 
analyst,  using  his  knowledge  and  intuition,  to  first  make  some 
exploratory  simulation  runs  to  get  the  search  into  the  "right  ball 
park,"  so  to  speak,  and  then  use  the  OPTIMIZER  to  provide  a  mathe¬ 
matically-disciplined  search.  Since  the  MOE  values  produced  by  the 
analyst's  computer  run  showed  that  the  most  progress  was  made  during 
the  first  11  simulation  runs  and  that  further  progress  was  more 
gradual,  the  OPTIMIZER  search  was  started  from  the  point  corresponding 
to  the  analyst's  run  No.  11.  This  time  the  OPTIMIZER  located  a 
maximum  MOE  value  on  run  No.  36  (i.e.,  on  the  OPTIMIZER'S  26th  run) 
and  indicated  at  run  No.  50  that  this  was  the  best  it  could  do.  It 
should  be  noted  that  the  analyst's  MOE  values  never  surpassed  this 
value  and,  in  fact,  did  not  come  close  until  run  No.  67.  (See  Figure 
4.) 

In  summary,  the  OPTIMIZER  functioned  best  as  part  of  a  complementary 
two-stage  optimization  process  which  coupled  the  knowledge  and  intuition 
of  the  analyst  with  the  mathematically-disciplined  search  of  the  OPTIMIZER. 
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Run  Number 

Figure  4  -  OPTIMIZER  and  Analyst  Run  Histories  (OPTIMIZER  Search  No. 3) 
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5.  Overview  of  Analyst1 s  Methodology 

The  apparent  superiority  of  the  analyst  over  the  OPTIMIZER  led 
us  to  look  briefly  into  the  procedures  used  by  the  analyst  in  solving 
this  particular  problem.  Such  an  investigation,  it  was  hoped,  would 
provide  insight  into  human  problem-solving  methods  that  could  be 
automated.  This  section  discusses  some  of  the  specific  techniques 
the  analyst  used  in  attempting  to  solve  this  particular  problem. 

The  human  analyst  solves  optimization  problems  by  developing  a 
strategy  to  explore  the  cause-and-ef feet  relationships  between  the 
variables  and  the  MOE.  The  major  characteristic  of  an  optimization 
problem  that  prevents  the  analyst  from  adopting  an  efficient  search 
strategy  is  usually  the  limit  on  the  number  of  variables  he  can 
manipulate.  Even  if  the  number  of  variables  is  small,  the  causal 
relationships  may  be  in  an  unmanageable  or  unrecognizable  form.  In 
either  case  the  first  task  of  an  analyst  is  to  reduce  the  problem  to 
a  more  workable  form.  Many  methods  can  be  used  to  accomplish  this. 

Three  such  reductions,  which  lowered  the  number  of  variables  from  29 
to  four,  were  actually  applied  while  solving  this  test  problem. 

The  first  reduction  involved  the  speeds  of  the  first  and  second 
escorts.  The  MOE  to  be  maximized  was  the  product  of  the  probability 
of  detection  and  the  average  range  of  detection.  The  probability  of 
detection  could  be  increased  if  more  area  per  unit  time  were  covered 
by  an  escort's  sonar.  An  increase  in  the  speed  of  an  escort  would 
allow  this  greater  coverage  if  the  performance  of  the  sonar  were  not 
downgraded  by  this  increase  in  speed.  A  calculation  was  made  which 
determined  that  there  would  be  almost  no  downgrading  of  sonar  performance 
between  the  minimum  and  maximum  allowed  speeds  for  an  escort.  Hence 
the  two  speeds  were  set  to  their  maximum  values  and  were  no  longer 
manipulated.  Actual  simulation  runs  were  made  to  verify  the  expected 
improvement  in  the  value  of  the  MOE  resulting  from  this  simplification. 

8.  Mandelbaum,  Jay;  NSRDC  CMD  Technical  Note,  CMD-42-72,  "Methodology 
Used  in  Solving  a  Test  Problem  Designed  for  Evaluating  the  HRB  Singer 
Optimizer  for  Large-Scale  Computer  Simulation  Models,"  September  1972 
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The  second  reduction  involved  equalizing  the  distances  from  the 
task  force  to  the  centers  of  the  first  and  second  patrol  areas.  There 
was  no  clear  advantage  in  having  a  difference  between  these  distances 
and  consequently,  three  simulation  runs  were  made  to  specifically  test 
whether  any  advantage  existed.  Since  the  results  showed  a  lower  value 
of  the  MOE  when  such  a  difference  existed,  one  of  the  distances  was 
eliminated  as  an  independent  variable. 

The  third  reduction,  by  far  the  most  substantial,  involved  the 
six  ranges  and  six  bearings  from  the  centers  of  the  first  and  second 
patrol  areas  which  defined  points  to  be  traversed  in  a  fixed  order. 

These  points  formed  the  escort  patrol  patterns.  The  analyst  had  no 
intuitive  grasp  of  the  effect  on  a  patrol  pattern  of  varying  ranges 
and  bearings  of  these  points.  He  did,  however,  have  an  intuitive  grasp 
of  the  general  type  of  patrol  pattern  that  should  be  used,  that  some 
type  of  zig-zag  pattern  would  give  the  best  results.  Hence  the 
variation  of  ranges  and  bearings  was  reduced  to  the  variation  of  zig¬ 
zag  type  patterns.  For  such  patterns,  the  variation  took  the  form  of 
changing  the  width  of  the  area  traversed.  Thus  the  24  ranges  and 
bearings  were  reduced  to  only  two  variables,  a  width  for -the  first 
patrol  area  and  a  width  for  the  second  patrol  area.  With  the  specifi¬ 
cation  of  a  zig-zag  pattern  and  a  designated  pattern  width,  the  12 
ranges  and  bearings  for  each  patrol  area  were  completely  defined  and 
therefore  removed  as  independent  variables. 

The  final  four  controllable  variables  remaining  after  the  preliminary 
simulation  runs  were  the  range  to  the  center  of  the  patrol  areas,  the 
bearing  of  the  first  patrol  area  from  the  task  force,  and  the  widths  of  the 
first  and  second  patrol  areas.  These  variables  were  manipulated  to  find  an 
optimal  value  of  the  MOE.  One  critical  fact  must,  however,  be  stated: 
an  optimal  solution  to  the  four-variable  problem  is  not  guaranteed  to 
be  an  optimal  solution  to  the  29-variable  problem.  Simulation  itself 
is  an  heuristic  process.  The  simplifications  which  reduce  the  number 
of  variables  to  a  workable  form,  when  simulating  fcr  an  optimal  solution. 


14 


are  also  done  heuristically.  The  object  is  thus  not  to  find  a  true 
optimal  solution  but  rather  to  obtain  in  the  most  efficient  manner  a 
value  of  the  MOE  judged  satisfactorily  close  to  the  optimal  value. 

After  the  reduction  of  variables,  the  next  step  in  the  development 
of  an  efficient  search  strategy  is  partitioning  the  problem  into  sub¬ 
problems.  This  partitioning  is,  in  general,  also  an  heuristic  process 
since  usually  one  cannot  be  sure  that  each  subproblem  is  independent  of 
all  others.  For  the  specific  test  problem,  some  extreme  cases  were 
run  to  permit  establishment  of  an  approximate  initial  value  for  the 
distance  to  the  center  of  the  patrol  areas.  Runs  were  then  made, 
which  sometimes  varied  the  widths  of  the  patrol  areas  and  sometimes 
varied  the  bearing  of  the  first  patrol  area,  in  order  to  improve  the 
value  of  the  MOE. 

Once  the  subproblems  have  been  defined,  one  final  component  of 
the  search  strategy  remains  to  be  specified:  the  decision  criteria 
used  to  change  the  variables  from  run  to  run.  The  governing  criterion 
is  that  the  change  must  increase  the  value  of  the  MOE.  However,  it 
may  not  be  possible  to  determine  before  making  a  run  that  such  an 
improvement  will  result.  In  such  a  case  it  may  be  useful  to  define  a 
related  MOE  in  such  a  way  that  it  will  exhibit  the  same  behavior  as 
the  actual  MOE  for  given  conditions  but  will  more  readily  provide  an 
indication  of  whether  the  change  will  improve  the  value  of  the  real 
MOE.  However,  whether  or  not  the  related  MOE  exhibits  the  proper 
behavior  usually  can  only  be  assumed.  Even  when  the  proper  behavior 
is  exhibited,  the  question  as  to  whether  the  value  of  the  real  MOE 
will  be  improved  may  be  answered  incorrectly.  The  difficulty  lies 
in  the  fact  that  the  selection  of  a  related  MOE  must  be  done  heuristically. 
One  cannot  be  sure  that  a  change  in  the  controllable  variables  will 
produce  the  desired  effect  on  the  true  MOE  even  if  the  expected  change 
does  occur  to  the  related  MOE. 

The  analyst's  solution  of  the  test  problem  contained  two  examples 
of  use  of  related  MOE's.  Two  graphs  (Figure  5)  were  made  for  each  run 
(each  "run",  involved  100  iterations  of  the  problem),  one  graph  for 
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Figure  5  -  Relative  Detection  and  No-Detection  Graphs 


those  iterations  in  which  detections  occurred,  and  the  other  for  those 
iterations  in  which  detections  did  not  occur.  The  relative  position  of 
the  submarine  with  respect  to  the  task  force  at  the  end  of  each 
iteration  was  plotted  as  a  point.  Clusters  of  points  on  the  no¬ 
detection  graph  indicated  where  sonar  coverage  was  weakest.  The 
related  MOE  was  the  density  of  the  clusters  of  points.  High-density 
clusters  implied  holes  in  the  screen  and  hence  the  patrol-area  widths 
needed  to  be  increased. 

At  one  point  in  the  analyst's  solution  of  the  test  problem,  there 
were  two  different  candidate  zig-zag  type  patrol  patterns.  One  run 
was  made  with  each  pattern  keeping  the  four  variables  at  the  same 
values.  The  detection  graphs  indicated  that  there  were  more  detections 
at  larger  ranges  in  one  of  the  two  cases.  Here  the  related  MOE  was 
the  percentage  of  detections  beyond  the  distance  to  the  center  of  the 
patrol  areas,  and  the  analyst  thus  chose  the  zig-zag  pattern  yielding 
detections  at  the  larger  range. 

With  the  development  of  the  related  MOE's,  the  analyst's  strategy 
was  complete.  He  was  able  to  optimize  the  reduced  problem  intelligently. 
One  additional  factor  was  involved  in  the  selection  of  related  MOE's: 
since  information  about  their  behavior  was  constantly  being  accumulated 
as  runs  were  being  made,  uncertainty  associated  with  them  was  reduced. 

The  only  observed  limitation  in  the  analyst's  method  was  the  inability 
to  deal  effectively  with  several  subproblems  simultaneously. 
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6.  Implications  for  Semi -Automated  Problem  Solving 


Within  the  context  of  solving  simulation-optimization  problems, 
four  levels  of  approaches  to  a  solution  can  be  defined.  These  levels 
range  from  a  completely  deterministic  solution  procedure  to  the 
traditional  human  analyst  approach. 

The  lowest  level  (Level  1)  approach  consists  of  a  series  of 
deterministic  manipulations  of  the  variables,  each  followed  by  a 
simulation  run,  until  the  MOE  is  considered  maximized.  (See  Figure  6.) 
This  approach  is  impractical  for  most  applications,  since  deterministic 
solutions  do  not  exist  for  most  simulation  problems. 

The  Level  2  approach  involves  an  automatic  manipulation  of  the 
variables  much  as  the  OPTIMIZER  operates.  (See  Figure  7.)  By 
monitoring  a  measure  of  effectiveness,  an  heuristic  method  of 
manipulating  the  variables  and  running  the  simulation  is  automatically 
selected  (the  selection  process  itself  is  also  heuristic)  and  applied. 
This  process  continues  until  an  answer  is  determined.  This  approach 
has  the  advantage  of  looking  at  all  the  variables  at  once,  of  not 
requiring  subproblem  definition,  and  of  obtaining  a  solution  under 
severe  time  constraints.  The  disadvantage  is  that  the  automatic 
method  selection  process  may  not  respond  properly  or  efficiently  to 
the  sequence  of  simulation  results. 

The  Level  4  approach  is  the  entirely  manual  one  used  in  the 
analyst's  solution  of  the  test  problem  (see  Figure  8)  and  discussed 
in  Section  5. 

The  Level  3  approach  lies  between  the  automatic  OPTIMIZER  approach 
and  the  traditional  analyst  one.  (See  Figure  9.)  This  Level  3  approach 
appears  to  be  the  most  fruitful,  and  it  is  thus  recommended  that  such 
approaches  be  the  subject  of  further  research.  The  automatic  method 
selection  and  variable  manipulation  features  of  Level  2  should  be 
retained  to  eliminate  the  need  for  subproblems.  However,  provisions 
should  exist  for  dynamically  overriding  the  selected  method  with  one 
chosen  by  the  analyst  from  a  library  of  such  methods. 
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Figure  7  -  Level  2  Solution  -  Automated  Heuristic 
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Figure  9  -  level  3  Solution  -  Man/lia chine 
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This  method  library  should  be  expanded  by  cataloging  actual 
methods  employed  by  the  analyst  when  solving  problems  with  the  Level 
4  approach.  Examples  of  such  methods  can  be  drawn  from  the  test 
problem  solution  presented  in  this  paper.  Extremum  points  should  be 
run  to  eliminate  variable  combinations  that  might  otherwise  be  tested 
in  a  purely  automatic  approach.  Geometric  considerations  should  be 
applied  to  reduce  the  number  of  variables  and  hence  substantially 
reduce  the  number  of  possible  combinations  of  the  variables. 

The  case  discussed  in  this  paper,  in  which  the  OPTIMIZER  was 
given  the  analyst's  eleventh  run  as  a  start  point,  indicates  the 
possible  advantages  of  such  an  approach.  A  large  number  of  runs 
would  have  been  required  for  the  OPTIMIZER  to  find  such  a  start  point, 
and  it  is  quite  possible  that  the  OPTIMIZER  could  never  have  done  it. 
Once  the  start  point  was  determined,  however,  the  OPTIMIZER  rapidly 
moved  toward  its  solution.  Hence  the  suggested  Level  3  approach  takes 
advantage  of  the  strong  points  of  both  man  and  machine.  In  the  actual 
experiment,  the  man  was  used  to  converge  rapidly  to  a  fairly  good 
solution  and  then  the  computer  was  used  for  a  rapid  advance  to  a 
solution  which  is  closer  to  optimal. 

In  general,  the  power  of  the  automated  process,  especially  as 
it  applies  to  problem  solving,  lies  in  its  ability  to  extend  the 
capabilities  of  a  man,  not  to  replace  him.  It  is  the  conclusion  of 
the  authors  that  applications  exist  in  which  an  automated  problem 
solver  will  work  effectively  when  properly  teamed  with  a  man.  It  is 
also  considered  that  great  benefit  can  ensue  from  this  man-machine 
interface,  both  in  extending  the  capabilities  of  an  analyst  and 
especially  in  drastically  cutting  down  the  time  to  solution. 
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